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DIGITAL IMAGE PROCESSING METHOD HAVING AN EXPOSURE 
CORRECTION BASED ON RECOGNITION OF AREAS CORRESPONDING TO 
THE SKIN OF THE PHOTOGRAPHED SUBJECT 

BACKGROUND OF THE INVENTION 

5 Field of the Invention 

[0001] The present invention relates to a digital image processing 
method. The invention relates particularly, but not exclusively, to an image 
processing method of human subjects being photographed by portable image 
taking devices, particularly of backlit subjects and the following description is made 
10 with reference to this field of application for convenience of illustration only. 

Description of the Related Art 

[0002] As is well known, one of the main problems limiting 
photographic image quality involves the generation of sub-optimal photographs 
due to the wrong exposure to light of the photographed subject. 

15 [0003] This problem is particularly suffered in portable devices 

such as mobile phones, wherein several factors concur in obtaining photographs 
that are wrongly exposed: the smallness of the available optical device, the lack of 
a flash device and the like. Moreover, the portable device nature and the 
traditional use of the photographs produced therefrom, particularly linked to the so- 

20 called multimedia messaging services or MMS, cause the acquisition of 
photographs of the type shown in Figure 3. 

[0004] Although it is impossible to provide a precise definition of 
a correct exposure, since the exposure depends on the photographed subject as 
well as on the personal taste of the person looking at the photograph, it is however 

25 possible to state that, for "normal" subjects (and thus not considering extreme 
cases, like a snow-covered landscape whose correct acquisition would involve an 
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intentional photograph overexposure), a correct exposure is obtained when the 
main features of the photographic image are reproduced by using an intermediate 
grey level. 

[0005] In the image processing field several techniques for 
5 improving the tone quality of photographic images are well known, such as 
histogram equalization, grey-level slicing, and histogram stretching. 

[0006] Although advantageous under many aspects, these prior 
art techniques have several drawbacks mainly linked to the fact of being 
independent from the visual content of the photographed images. 
10 [0007] The article entitled "Automated Global Enhancement of 

Digitized Photographs" by Bhukhanwale et al., published on the IEEE Transaction 
on Consumer Electronics, vol. 40, no. 1, 1994, which is hereby incorporated by 
reference in its entirety, describes instead an algorithm being capable to identify 
visually important regions in a photographic image, by adjusting the image 
1 5 exposure so that these regions occupy intermediate tone levels. 

[0008] Moreover, the European patent application no. EP 
01830803.1 filed in the name of STMicroelectronics, the assignee of the present 
application, which is hereby incorporated by reference in its entirety, describes an 
algorithm being similarly capable to identify visually important regions in a 
20 photographic image in order to replace them at intermediate tone levels. This 
algorithm directly processes images of the Bayer Pattern type and simplifies the 
statistical measures used to detect regions in the image having a high information 
content, i.e., visually important regions. 

[0009] The algorithms provided in this document directly operate 
25 on the image in the Bayer Pattern format and they comprise the following steps: 
extraction of the Bayer Pattern green plane or channel G: this plane 
provides a good approximation of the luminance Y. 

[0010] visual analysis: once the channel G has been extracted, 
the visually interesting regions are identified on this channel. For this purpose, the 
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green plane is split into N blocks having the same size and the following statistical 
values are calculated for each block: 

[0011] focus: it characterizes the block sharpness and it is used 
for identifying the regions comprising high-frequency components, corresponding 
5 to details of the photographed image; 

[0012] contrast: it is related to the image tone range - the higher 
the contrast, the higher the insulation of the so-called clusters of points in the 
block, i.e., the higher the block visual impact. 

[0013] In order to obtain important visual features, independently 
10 from the lighting conditions of the photographed image, the visual analysis is 
performed on an image having an intermediate luminosity produced by making a 
temporary correction only based on the average value of the channel G calculated 
on the whole plane. The algorithms further perform exposure adjustment: once the 
visually interesting regions have been detected/the exposure adjustment is 
15 performed by using the average grey levels of these regions as reference values. 
In greater detail, the photographed image is changed so to bring the average value 
of these regions to a target value T by changing all the pixels belonging to the 
Bayer Pattern. This target value T should be a value ranging around 128 and it 
should take into consideration a possible correction range performed after the 
20 color reconstruction of the corrected Bayer Pattern. This means that, in certain 
cases, the target value T could be substantially lower than 128. 

[0014] To this aim, a simulated response curve of a digital image 
taking device or camera is used, schematically shown in Figure 1. 

[0015] This curve gives an evaluation of how the light values 
25 picked up by the camera are turned into pixel values, i.e., it represents the 
function: 

/(?) = / (1) 
q being the light amount and I the final pixel value. 
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[0016] This simulated response function (1) of a camera can be 
expressed in a parametric way: 



Kq) = 




(2) 



[0017] A and C being the control parameters of the curve shape 
5 and the value q being expressed in base 2 logarithmic units (also known with the 
name "stops". It is possible to evaluate these control parameters A and C by using 
the information comprised in the article by Mann et al. entitled "Comparametric 
Equations with Practical Applications in Quantigraphic Image Processing", IEEE 
Transactions on Image Processing, Vol. 9, no. 8, 2000, which is hereby 
1 0 incorporated by reference in its entirety. 

[0018] It is also possible to obtain experimentally the values of 
these parameters A and C or to set them in order to realize a particular final effect 
(for example, a more or less marked improvement of the contrast). In particular, 
Figure 1 shows the trend of the simulated response curve expressed by the 
1 5 formula (2) with A = 7 and C = 0. 1 3. 

[0019] By using this simulated response curve f and an average 
grey level avg for the visually important regions, the distance A of an ideal 
exposure situation is expressed as: 



20 and the grey value l(x, y) of a pixel with position (x, y) is thus changed in: 



It is worth noting that all the grey values of the pixels are corrected. 

[0020] In particular, the above-mentioned changes are 
substantially a look-up table (LUT) transformation (i.e., they can be put in a table in 
25 order to be then referred to) and Figures 2A and 2B show two different 



A = r 1 (128)-r 1 (avg) 



(3) 



/W) = /(TV(*.jO) + A) 



(4) 
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transformations (the curves LUT1 and LUT2) generated from a first simulated 
response curve f1 with values A = 7 and C = 0.13 and a second simulated 
response curve f2 with values A = 0.85 and C = 1 . 

[0021] It is worth noting that the distance or offset of the value 
5 128 is 1 .24 for f1 and 0.62 for f2 respectively (starting from a same input value 
equal to 72). 

[0022] From the Figures 2A and 2B it is evident that the first 
curve LUT1 has a more linear trend, while the second curve LUT2 has a so-called 
range trend. 

10 [0023] Although advantageous under several aspects, these prior 

art techniques are not very effective in the case of portable devices like mobile 
phones for which the photographic images are often backlit and they are mainly 
focused on human figures, when the user uses the image transmission for 
videophony, as shown in Figure 3. 

1 5 BRIEF SUMMARY OF THE INVENTION 

[0024] One embodiment of the present invention provides an 
image processing method having such features as to overcome the limits still 
affecting prior art techniques. 

[0025] One embodiment of the present invention detects the 
20 features in the photograph of the skin of the subject being photographed in order to 
select and find convenient interesting regions on whose base an exposure 
adjustment/correction is applied. 

[0026] One embodiment of the present invention is directed to a 
digital image processing method that includes: extracting chromatic information of 
25 an image taken by an image taking device and related to a human subject; 
detecting visually interesting regions in the taken image by recognizing areas 
corresponding to skin of the subject, wherein the recognized areas are the visually 
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interesting regions; and correcting exposure of the taken image by normalizing a 
grey scale of the taken image based on the visually interesting regions. 

[0027] The features and advantages of the method according to 
the invention will be apparent from the following description of an embodiment 
5 thereof given by way of non-limiting example with reference to the attached 
drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0028] In the drawings: 

[0029] Figure 1 shows the trend of a simulated response curve of 
10 a known image taking device; 

[0030] Figures 2A and 2B show LUT transformations related to 
different curves like the one in Figure 1; 

[0031] Figure 3 shows an image of a backlit subject taken by a 
known image taking device; 
1 5 [0032] Figure 4 shows an illustrative diagram of a step of the 

image processing method according to one embodiment of the invention; 

[0033] Figures 5A-5C and 6A-6C show following image 
processings for detecting important areas which are used in a step of the image 
processing method according to one embodiment of the invention; 
20 [0034] Figures 7A-7D schematically shows the method according 

to one embodiment of the invention by means of following image processings; 

[0035] Figure 8A shows an image of a subject; Figure 8B shows 
an image of the subject of Figure 8A with areas highlighted corresponding to the 
skin of the subject; and Figure 8C shows a detection histogram of the image of 
25 Figure 8A; 

[0036] Figures 9A-12B show processed images obtained by the 
method according to alternate embodiments of the invention. 
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DETAILED DESCRIPTION OF THE INVENTION 

[0037] An image processing method according to one 
embodiment of the invention performs an exposure correction of a digital 
photographed image taken by an image taking device on the basis of a recognition 
5 algorithm of the skin of the photographed subject, thus improving the final 

photographic image quality, in a decisive way in the case of backlit subjects as in 
Figure 3. 

[0038] In particular, the method comprises the following steps: 
[0039] 1 ) a first extraction step of chromatic information from the 
1 0 photographic image; 

[0040] 2) a second visual analysis step using a recognition 
method of the areas corresponding to the skin of the subject photographed in the 
photographic image; and 

[0041] 3) a third exposure adjustment step of the obtained 
1 5 photographic image. 

[0042] 1) Extraction Step 

[0043] As has been seen with reference to known image 
processing methods, the method provides the extraction of the green channel G of 
the image taken when the images are in the Bayer format. 
20 [0044] On the contrary, it provides the extraction of the luminance 

channel Y for images of the YcbCr type obtained from RGB images. 

[0045] 2) Visual Analysis Step 

[0046] This analysis can be performed on: 

[0047] 1 . images in the RGB format; 
25 [0048] 2. images in the Bayer Pattern format generating, from an 

initial image, a conveniently sub-sampled RGB copy. 

[0049] In particular, by using this skin recognition method, a 
plurality of visually interesting regions corresponding to the skin of the 
photographed subject are detected. 
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[0050] The chromatic information obtained during the first 
extraction step is thus used. 

[0051] In particular, using Bayer data, it is possible to operate on 
three color planes and on sub-samples having a size corresponding to a quarter of 
the initial data, as schematically shown in Figure 4, thus considerably reducing the 
calculation efforts of the method. 

[0052] 3) Third Exposure Adjustment Step 

[0053] This adjustment can be performed in two ways: 

[0054] 1 . correction of images in the RGB format; 

[0055] 2. correction of images in the Bayer Pattern format before 
a following color interpolation algorithm. 

[0056] In the case of the correction of images in the RGB format, 
once the visually important pixels have been detected as above-mentioned (i.e., 
the pixels belonging to the area corresponding to the skin of the photographed 
subject), a known exposure correction algorithm is used, wherein the average grey 
level of the known pixel clusters is considered as belonging to the skin of the 
photographed subject. 

[0057] In other words, the pixels belonging to the subject skin are 
placed at the intermediate level of the image grey scale and all the remaining 
image pixels are placed once again based on this average level. 

[0058] In particular, once the luminance value has been 
corrected from an original value Y to a revised value Y' that reflects the average 
grey level of the know pixel clusters corresponding to the skin of the subject, 
according to the above-mentioned steps (2), (3) and thus using the information 
comprised in the article by Sakaue et al. entitled "Adaptive Gamma Processing of 
the Video Cameras for the Expansion of the Dynamic Range", IEEE Transaction 
on Consumer Electronics, Vol. 41, n. 3, August 1995, which is hereby incorporated 
by reference in its entirety, starting from a curve of the type shown in Figure 1 , the 
pixel chromatic values can be reconstructed according to the formulas: 
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r =o.5-f~(/?+y)+/?-y 



(5) 



fv> 



G = 0.5 



_.(G+r)+G-yJ (6) 



5 = o.5 ^y(5+y)+5-yj (7) 

R, G, B being the color values of the input pixels. 

[0059] In the case of the correction of images in the Bayer format 
the formulas (5), (6) and (7) cannot be used and the output product will be 
5 obtained by simply applying the relation (4) to all the pixels of the pattern. 

[0060] The recognition method of the areas corresponding to the 
skin of the subject photographed in the photographic image will be now described 
in greater detail. 

[0061] Several recognition methods of the color of the skin of the 
10 photographed subject are known, substantially based on the application of a 
threshold to a color probability measure for the skin. 

[0062] In fact, the colors of the human skin belong to a particular 
color category, different from the colors of most natural objects. In particular, in 
the article by Zarti et al. entitled "Comparison of Five Color Models in Skin Pixel 
15 Classification", Proc. Of Int. Workshop on Recognition, Analysis and Tracking of 
Faces and Gestures in Real-Time Systems, IEEE Computer Society, Corfu, 
Greece, pages 58-63, 1999, which is hereby incorporated by reference in its 
entirety, it has been shown that the colors of the human skin are clustered and that 
the skin changes between each subject are substantially due to a different intensity 
20 and they can thus be reduced by using only the chrominance component of the 
subject image. 

[0063] Moreover, in the article by Yang et al. entitled "Skin-Color 
Modeling and Adaptation", Technical Report CMU-CS-97-146, School of Computer 
Science, Carnegie Mellon University, 1997, which is hereby incorporated by 
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reference in its entirety, it has been shown that the human skin color slicing can be 

represented by a bidimensional Gaussian function in the chrominance plane. The 
center of this slicing is determined by the mean vector and the amplitude of the 

bell thereof is determined by the covariance matrix I, these two values being 
5 evaluated starting from a convenient group of test data. 

[0064] The conditional probability p( x |s) of a pixel block to 
belong to a human skin color class starting from the chrominance vector thereof x 
is thus given by: 

^ (8, 

10 d (*) being the so-called Mahalonobis distance of the chrominance vector x of the 
mean vector ^ , defined as: 

[d(x)f =(x-fLyl- l (x-fl) (9) 

In other words, the value of the Mahalonobis distance d(x) of a pixel block with 

chrominance vector x determines the probability of this block to belong to a 
15 predetermined human skin color class. The higher the Mahalonobis distance d(x) 

is, the lower the probability of the block belonging to this human skin color class. 

[0065] Given the considerable amount of color types, of distance 
measures and of bidimensional slicings, a considerable variety of image 
processing algorithms can be considered. Moreover, the lighting conditions and 
20 the color models can change according to the image taking modes. 

[0066] Advantageously, the method comprises a step of 
recognizing a portion of the photographic image corresponding to the subject skin. 

[0067] In a first embodiment, this recognition step is substantially 
based on a probabilistic function. 
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[0068] In particular, for each pixel of an image taken in the YcrCb 
format a probabilistic slicing is prepared to evaluate if this pixel must be classified 
as belonging to the subject skin. Based on this slicing, a new image is thus 
processed with a normalized grey scale, wherein the subject skin is highlighted as 
5 indicated in Figures 5A-6C on two different photographic images which depict 
human subjects. 

[0069] Based on this first embodiment of the recognition step of 
the photographed subject skin, the image pixels with higher grey values are 
considered as belonging to the skin of the photographed subject. 

10 [0070] The areas being detected by using this first embodiment 

of the recognition step of the skin of the photographed subject on an image shown 
in Figure 5A are shown in Figure and 5C. Similarly, Figure 6C highlights areas of 
the image of Figure 6A corresponding to the skin of another photographed subject, 
according to the first embodiment of the recognition step. 

15 [0071] In a second embodiment, the recognition step of the 

image areas corresponding to the skin of the photographed subject is substantially 
based on a single threshold area and it provides an RGB-format image processing 
in order to produce a chrominance slicing histogram starting from normalized 
channels r and g as described in the article by Soriano et al. entitled "Skin Color 

20 Modeling Under Varying Illumination Conditions Using the Skin Locus for Selecting 
Training Pixels", Real-time Image Sequence Analysis (RISA2000, August 31 -Sept. 
1, Finland) , which is hereby incorporated by reference in its entirety. In particular, 
the normalized channels r and g are defined as: 

r= R/(R+G+B) (10) 

25 g = G/(R+G+B) (11) 

The resulting bidimensional histogram shows the chrominance slicing in the image 
and the areas having the right human skin chrominance slicing are thus detected 
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by applying a single threshold area. In particular, the pixels of the processed 
image belonging to the threshold area are classified as belonging to the subject 
skin. 

[0072] Similarly, Figure 5B shows the areas, detected by using 
5 this second embodiment of the recognition step, corresponding to the skin of the 
photographed subject from an image shown in Figure 5A of a human subject. 
Also, Figure 6B highlights areas of the image of Figure 6A corresponding to the 
skin of another photographed subject, according to the second embodiment of the 
recognition step. 

1 0 [0073] Figure 7A-D schematically show the following processings 

of an image concerning a human subject, particularly backlit, after the different 
steps of the method according to alternate embodiments of the invention. 

[0074] In particular, on a taken image (Figure 7A) the recognition 
step of the areas corresponding to the photographed subject skin is performed with 

1 5 a probabilistic (Figure 7B) or threshold (Figure 7C) method. 

[0075] An exposure correction step can thus be performed by 
using the areas detected as belonging to the skin of the photographed subject in 
order to normalize the grey levels obtaining a final processed image (Figure 7D), 
the best image quality being immediately evident by comparing it with the starting 

20 image (Figure 7A). 

[0076] Advantageously, the regions being detected in the 
recognition step as belonging to the subject's skin are used as visually important 
images for the following exposure adjustment step of the photographic image. 

[0077] In a preferred embodiment of the image processing 

25 method according to the invention, the recognition step of the areas belonging to 
the photographed subject's skin processes a .8-bit image of the Bayer type, 
constructing a color image of sub-samples with size corresponding to a quarter of 
the initial data, obtained as previously described and schematically shown in 
Figure 4. 
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[0078] Starting from this color image of sub-samples, a 
recognition step of the areas belonging to the photographed subject's skin is 
performed using a chrominance slicing histogram according to the first probabilistic 
embodiment or the normalized channels r and g according to the second threshold 
5 embodiment. 

[0079] However, in this second case, the normalized channels r 
and g are defined as: 

r=R/(R+G*+B) (12) 
g = G7(R+G*+B) (13) 

10 being 

G*=(G1+G2)/2 (14) 

The resulting bidimensional histogram shows the chrominance slicing of the 
processed image, therefore the areas corresponding to the photographed subject 
skin, as schematically shown in Figures 8A-C, showing in series a taken image of 

15 the Bayer type (Figure 8A), the image (Figure 8B) processed to detect the areas 
corresponding to the photographed subject skin and a detection histogram (Figure 
8C) r-g of these areas. 

[0080] The method finally comprises a reconstruction step of the 
color of the image taken according to the relations (5) to (7), already shown with 

20 reference to the prior art, R, G, B and R', G', B' being the red, green and blue 
values of the images being respectively taken and processed. 

[0081] The step sequence being described is suitable for a 
simple change allowing a correction to be performed directly on images in the 
Bayer Pattern format in favor of a further simplification from the calculation point of 

25 view. In fact, once the image for the skin detection according to the diagram of 
Figure 4 has been constructed, the average value calculated for the regions 
concerned can be used to directly perform the Bayer Pattern correction, using for 
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example the modes described in the above-mentioned European patent 
application no. 01830803.1 

[0082] It is however worth noting that the color reconstruction 
formulas described in the equations (5), (6), (7) cannot be used in this case and 
5 the output product of the corrected Bayer Pattern will be obtained by simply 
applying the relation (4) to all the model pixels. 

[0083] In other words, the grey value l(x, y) of a pixel with 
position (x, y) is modified in: 

r{x 9 y) = f(f' l (I{x 9 y)) + A) t (15) 

10 where A is the distance of the ideal exposure situation as expressed in relation (3). 
The image processing of a backlit subject being performed by using a CMOS-VGA 
sensor and an evaluation kit on the Windows® platform is shown in Figures 9A-9B, 
wherein in the panel V the areas detected as belonging to the photographed 
subject's skin have been indicated on a black background. 

15 [0084] Similarly, Figures 10A-12B show the results of a 

simulation of the method performed by the inventors starting from images taken by 
a common VGA sensor in the compressed jpeg format (Figures 1 0A, 11 A) and by 
a 4.1 Mpixel CCD sensor of a traditional average-band DSC (Digital Still Camera) 
(12A) and the images processed with the method 10B, 11B, 12B respectively have 

20 been indicated, wherein the image qualitative improvement is completely evident. 

[0085] All of the above U.S. patents, U.S. patent application 
publications, U.S. patent applications, foreign patents, foreign patent applications 
and non-patent publications referred to in this specification and/or listed in the 
Application Data Sheetare incorporated herein by reference, in their entirety. 

25 [0086] From the foregoing it will be appreciated that, although 

specific embodiments of the invention have been described herein for purposes of 
illustration, various modifications may be made without deviating from the spirit 
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and scope of the invention. Accordingly, the invention is not limited except as by 
the appended claims. 
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